Workflow of Write Operations in GFS

We’ve discussed how GFS carries out file operations, and we have seen the two of them (create and read files) in the previous lesson. In this lesson, we will look into what kind of write operations GFS supports and the workflow of these operations.

Writing data to the file#

Data can be written at a random position in the file or appended to the file in case of sequential writes. GFS offers two operations–a random write and an append operation. In a random write operation, the client provides the offset at which the data should be written. In an append operation, the data is written at the end of the file at the GFS chosen offset.

All write operations from the clients are carried out by the replica that has the current lease, called the primary replica. The primary replica then forwards the request to secondary replicas and replies to the client once the request has been processed. Since it is the manager who decides which replica will take the lease, therefore, replicas don't need any election algorithms to choose a primary between them.

Note: Leases are a fault-tolerance mechanism that enable the manager to provide good availability under different kinds of failures. Lease value should not be either too large (in which case if a server takes a lease and dies, manager might need to wait out for the lease to expire before it can give it to some other server) or too small (in which case manager might get excessive traffic for lease renews). A good lease value usually depends on specific use cases and evolves with the operational experience with the system.

Serialization is required to cope with the data inconsistency issue among the replicas shown above. It is challenging to serialize write operations across multiple replicas. The manager has to do a lot of work to manage all this. So, the manager uses the lease mechanism to reduce this management overhead. With a lease, all the write operations are carried out on a single replica at a time, which makes it easy to serialize operations on a single chunkserver. Let's see how it works by looking into the workflow of a write operation.

Workflow of write operations#

The workflow for both the random write and the append operation is almost the same and is illustrated below. The only difference is the chunk to which the data is written.

• In random writes, the clients provide the offset at which the data is to be written. The client can map the offset to a chunk index if each chunk contains all 64 MB (the maximum data a chunk holds/the size of the chunk) and there is no padding. This is internal to GFS. It is also possible that it uses the byte range instead of an index to deal with the padding. The chunk is found using the client-provided offset.

• In append operations, the client doesn't need to provide the offset. The data is pushed to the last chunk.

The edge cases#

The append operation writes data to the end of the file. To perform the operation, the client asks the manager for the last chunk's metadata (chunk handle, and chunkservers). The manager node looks into the metadata for the last chunk of the file and responds to the GFS client with the required metadata. The client then sends data to the chunkserver holding the last chunk. There are three different scenarios the client has to deal with depending on the available space on the last chunk and the total size of the data to be written. Let's look at each scenario.

1. Suppose the last chunk has available space for appending new data. In that case, the chunkservers write that data in the specific chunk returned by the manager and respond to the client with a success message, as shown in the following illustration.

Since each chunk is replicated on multiple chunkservers for availability, the primary replica has to ensure that the write operation is performed on all replicas.

2. If the last chunk already has 64 MB of data, then the chunkserver responds to the client that it is already full. The client then asks the manager to create a new chunk. The manager generates a new chunk ID, allocates the chunkservers where this chunk's data would be placed, and responds to the client with the new chunk's metadata. The client will then write the data to the new chunk. An example is shown in the following illustration.

One approach is for the chunkserves to inform the manager about the full chunk in the last heartbeat message to save the client a round-trip to the manager later on. On the other hand, delaying such a declaration to the manager might reduce the immediate load on the manager.

3. It is possible that the last chunk is not full, but doesn't have the capacity to accommodate all data bytes in the append request. In this case, the chunkserver holding the last chunk will respond to the client with a message that the available space in that chunk is less than the size of the write. The chunkserver likely adds information about how much available space it has on the last chunk, based on which the client will split the writing data into two. The first part of the split will be written to the end of the last chunk, and for the second part, the client asks the manager to generate a new chunk. The remaining data will be written at the start of the newly created chunk.

Decoupling the control and data flow#

Decoupling the control and data flow helps efficiently use the network. We’ve seen in the workflow of a write operation that the control flows between the client and the manager, from the client to the primary replica, and from the primary replica to the secondary replicas. On the other hand, data flows linearly from the client to the chunkservers. Each machine forwards data to the nearest machine on the network that has not yet received it. It does this to maximize the usage of each machine's network bandwidth.

Note: The network distance depends on a specific network topology (the way servers are connected in a network). Near might mean two servers in the same rack or minimizing the number of switches that must be crossed to reach each other. Usually, when we need to traverse more switches to reach the destination, the bandwidth might become less, and latency might increase.

The following illustration shows the control versus data flow.

The linear data flow through a chain of chunkservers over a TCP connection avoids going through network bottlenecks and high-latency links while minimizing the latency to push all data.

Summary#

In this lesson, we learned how GFS handles write operations. We have also seen some edge cases that come up when writing and how those are handled. In the end, we highlighted the virtues of separating control and data flow. The next lesson will cover two further operations—the deletion, and the snapshot.